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ABSTRACT 



This paper illustrates several collaborative program 
evaluation models and designs used in Newark, New Jersey, public schools 
since a state takeover in 1995. As part of its comprehensive design for 
school reform, the school district developed collaborations with community 
partners to provide direct services to schools. Each evaluation combined 
varied data and mixed methods across the research process. Using qualitative 
and quantitative methods allows for multiple lenses and contributes to 
meaningful tracking of program implementation, exploration of program models, 
and validations of important program outcomes. The research process in all 
studies involved multistage designs with formative and summative components, 
varied partnership participation and perspectives, and flexibility in 
planning. The programs evaluated are: (1) Principals' Leadership Institute, a 

professional development program for teachers; (2) Project Grad-Newark, a 
program focusing on academic success through whole school improvement; (3) 

New Beginnings Kindergarten Program, an effort to restructure kindergartens; 
(4) School Based Health Care Clinics; and (5) 21st Century Community Learning 
Centers, extended-day academic and enrichment programs. These evaluations 
show how exploratory and confirmatory investigations using multiple 
techniques can strengthen design, analysis, and inference, as well as improve 
conceptual clarity, flexibility, comprehensiveness, scientific rigor, and 
efficiency. One appendix contains diagrams of collaboration models, and the 
other contains examples of program evaluation products. (Contains 15 
references.) (SLD) 
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Multiple Voices and Mixed Methodologies to Support Comprehensive 

School Reform 



Introduction: 

This paper will illustrate several collaborative program evaluation models and 
designs, which have been conducted in Newark Public Schools since a State takeover 
of its 82 schools in 1995. As part of its comprehensive plan for school reform, the 
district, which is the largest in the State, prioritized the development of collaborations 
with community partners to provide a variety of direct services to schools. With the 
changing nature of educational reform more comprehensive in scope, and represented 
by broader-based community participation and partnerships, evaluation paradigms have 
also changed and begun to look very different. With external partnership funding 
currently over 10 million dollars annually and decentralized participatory management 
structures taking place at the.local school level, community/school collaborations have 
an increasingly significant impact on the district's education planning, on program 
decision making and on the nature of evaluation paradigms. 

The practical realities, which large urban public schools are driven by, are often 
different from the research, academic, theoretic, and service goals of universities, 
healthcare institutions, corporate entities, and non-profit community based 
organizations. Evaluation outcomes and the effectiveness of school based initiatives 
which represent collaborative partnerships can serve many purposes. 

School districts are most likely to undertake the evaluation of educational 
initiatives to i) determine efficacy of reform, ii) inform program planning and staff 
development, and to iii) advocate for the financial sustainability of grant funded 
initiatives. Foundations, corporate sponsors, and community-based organizations may 
use formative evaluation data as indicators of their public service, their investments and 
validation for their funding focus. However, the collaborative nature of modern school 
reform has brought together stakeholders with varied perspectives and interests, 
thereby challenging the need to find common ground that will better inform program 
planning, stakeholder interests, and evaluation paradigms. Structural differences of 
stakeholders no longer provide excuses for declaring effective collaborations to be 
insurmountable. In fact, partnerships through the reform agendas are contributing to 
promising and innovative practice models with new vision and broader expertise. A 
chart, which illustrates the Newark Collaboration Model, is included in Appendix A. 

Each evaluation, which is discussed in the paper, combined varied data and 
mixed methods across the research process. Although the research community 
continues to be influenced by "vestiges of the paradigm wars, this paper supports that 
qualitative and quantitative research methods free researchers to use a variety of data 
and methods which provide multiple lenses and contribute both to meaningful tracking 
of program implementation, exploration of program models, and validations of important 
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program outcomes (Tashakkori and Teddlie, 1998). Such techniques not only serve to 
better inform the contexts within which educational change occurs, allow for the analysis 
of data related to program outcomes, provide structures and timelines to examine long 
term impact, but also help to cross validate and identify issues about the rigor of the 
methodology and the reliability of findings. This paradigmatic approach evolves from 
the complex nature of the studies discussed in the paper, rather than from being 
imposed a-priori by the researcher. A chart which illustrates the Multiple Method 
Paradigm is included in Appendix A. 

Mixed method evaluation, in combining the use of quantitative and qualitative 
techniques, allows the researcher to more effectively evaluate educational programs 
that are characterized by complexity. This approach ultimately increases the validity 
and reliability of evaluation data as the evaluator is able to capitalize on the strength of 
each technique, while at the same time, minimizing the weaknesses that are frequently 
associated with reliance on a single method (Frechtling, 1997). The combination of 
multiple methods within a given design also affords the evaluator the opportunity to 
revisit the design based on information yielded from the use of more than one method. 
Thus, the evaluator may, on the basis of focus group data gathered during the earlier 
stages of the evaluation, modify or expand upon the initial design. 

The use of mixed method evaluation within the context of collaborative 
arrangements involving evaluator and stakeholders, tend to foster more inclusive buy-in, 
and improve access to multiple data sources that inform program as well as student 
outcomes. The combination of both benefits, often result in more cost effective 
evaluation designs. This paper presents the designs associated with several evaluation 
projects which illustrate the strength of incorporating multiple voices and mixed 
methodologies within program evaluations. 

Data Sources, Process and Issues: 

The research process in all studies involved 1) multi-stage designs with both 
formative and summative components, 2) varied partnership participation and 
perspectives across the research stages, and 3) flexibility in planning, such as changing 
timelines and evolving evaluation designs. These requirements help to combine the 
"high stakes" nature of measuring student achievement outcomes with the meaningful 
contributions of formative inquiry. They also facilitate the examination of process as 
well as analyses of multilevel data. Evaluations of educational reform initiatives are 
currently challenged to examine whether programs improve educational outcomes at 
various levels, i.e. the student, the classroom, the school, and across multiple sites. In 
light of these issues, the practical research endeavors must accommodate strategies 
related to availability and access of appropriate data, remain faithful to principles of 
scientific rigor to report accurately and reliably program outcomes, as well as capture 
the dimensions of change, and the varied contexts within which that educational change 
is occurring. They must also allow us to elaborate upon the role of partnership in 
building capacity to support the District’s whole school reform process. 
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Principals* Leadership Institute - 



The Newark Public School District research team led the evaluation to determine 
long-term impact of a professional development program for school leaders involving 
the collaboration of 8 universities and funded by 3 foundations (Walker, Mitchel, and 
Turner, 1999). The evaluation design included the application of a mixed method 
approach in which extensive qualitative data was gathered on the pedagogical structure 
of the institute, the varied theoretical assumptions inherent in the training and the nature 
of the professional development activities. Satisfaction with the professional 
development experience was measured through survey techniques based on a panel 
design. Ridit analysis was used to compare the data obtained from the three sets of 
surveys administered to principals and vice-principals. This design allowed for the 
measurement of programmatic changes on the same sets of attitudes at different times. 
These findings are presented in Appendix B. 

The evaluation of this project necessitated a close working relationship between the 
evaluator, the director of the professional development institute and university partners. 
The working relationship between all three was instrumental for framing the salient 
issues that needed to be addressed in the evaluation. The thoughtfulness and 
candidness of survey data from University partners' and principals provided empirically 
based evidence with which to examine the alignment of varied professional 
development models with principal expectations and needs based on their daily 
experience of reform exigencies. - i.e. raising critical issues about the degree of 
congruence or lack thereof, between the practical experiences of principals and the 
efficacy of various models grounded in very different theoretical premises. 

The different theoretical postures adopted by each of the eight universities created 
some challenges for the evaluation. First, charting administrators' reaction to the 
professional development program was confounded by the fact that their experiences 
were dissimilar both in terms of "content* and pedagogical delivery. Second, even 
within the same school, because of the random assignment to university cohorts, any 
given school's administrative team could have been exposed to three or four widely 
varying experiences. In order to work around these difficulties, the evaluation first 
called for intensive interviews with each of the eight universities. From the interviews 
we were able to construct profiles of each university staff development model along the 
following dimensions: (i) its underlying theoretical premises regarding leadership; (ii) 
the kinds of professional development activities derived from these premises; (iii) the 
universities' self-definition of their roles in the program; and (iv) their expectations for 
participants. 

School-based administrators on the other hand, were asked to rate their 
experiences within their cohorts from several different perspectives. They were also 
asked through a series of open-ended questions to provide evaluative comments on the 
program. The administrators' comments were subsequently linked with their respective 
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university responses. Conjoining the administrators' responses with the qualitative data 
gleaned from the interviews conducted with the universities enabled us to more clearly 
delineate the differential impact of each university's model on participants' rating of the 
effectiveness of the endeavor. 

Prnjfict Grad-Newark - 

The Ford Foundation and Lucent Technologies fund this program to the tune of 
10 million dollars. Project Grad focuses on a feeder school cluster of schools to 
provide a critical mass of academic supports from pre-k through 12*^ grade with the 
long term goal that every student succeed academically, graduate high school and 
pursue higher education. The program was first established in Houston Texas and 
both foundations view it as a critical funding component of their overall commitment 
to urban school reform. There is an interest, if proven successful, to replicate the 
model on a national level. The foundations have hired a private consulting firm to 
track the impact of the program. However, the evaluation planning, which is 
comprehensive in nature has included the consulting firm, the project administrator 
and the district research team. Partnership buy-in facilitated support for the multi- 
faceted and cohort tracking nature of the project. 

Partnership also enabled a division of labor and responsibilities for accessing 
primary and secondary sources of data, again using multiple method design and 
strategies. Major components of the program focus on improving student 
achievement , school and classroom environments, discipline, providing additional 
student supports such as tutoring, and a college scholarship goal incentive 
program. The underlying assumption of the model is whole school improvement. 
Evaluation planning and prospectus include the collection of multi years of project 
Grad school trends on both student and school level measures such as test scores, 
attendance, graduation rates, and school climate. Both individual level and 
aggregate school-level data will be collected as available on student and school 
outcomes on achievement measures such as state assessment tests at benchmark 
grades 4, 8, and 1 1 and other district standardized tests. 

Finding creative alternatives for research designs is key in light of practical 
realities and limitations, such as the absence currently of a sophisticated and costly 
central data management system that would make possible the ongoing and 
systemic longitudinal tracking of student data. Analysis of data for the Newark 
Project Grad will include comparisons of multi-year trends over a 4-year period for 
both project Grad and comparison schools. The years 1997-1998 and 1998-1999 
have provided baseline data with timelines for annual updates on the stages of 
program development, outcomes including longer range impact for high school 
graduates into the years 2002 and 2003, and identify issues of prograrn 
congruence, contributions and incorporation into systemic changes taking place at 
the school and district levels. 
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New Beginnings Kindergarten Program - 



New Beginnings has been part of an ongoing multi-year early childhood initiative 
to effect reform in the areas of curriculum, professional development, classroom 
management and student assessment. In 1996-1997, the first year of the New 
Beginnings initiative, the collaboration between Newark Public Schools and Bank Street 
College established 16 model kindergarten classes at 16 different schools. During the 
following 2 years, the model expanded to include over 100 pre-kindergarten to first and 
second grade classrooms. As the program expanded, a number of students in the 
original kindergarten cohort continued to receive New Beginnings support in grade 1 
and/or grade 2. The multi-year effort to restructure early childhood classes using the 
New Beginnings model continues to provide a network of resources and supports to the 
Newark early childhood program as the larger district restructuring and whole school 
reform process proceeds. Of three initiatives designed to improve kindergarten 
programs in the NPS, New Beginnings has been the most comprehensive. The other 
two are more limited in scope, and training. 

Evaluation Planning included Bank Street College Administration and research 
consultants, the Newark Public Schools Office of Early Childhood Education and the 
Office of Planning, Evaluation and Testing. Evaluation planning over three years 
reflects a multi-year collaborative effort between Bank Street College of Education and 
the Newark Pubic Schools to restructure Kindergarten and, in years two and three to 
expand the program to other early childhood classrooms, and with broader inclusion of 
administrators, parents, and teacher aides. 

An evaluation of the first year (Kopacsi and Hochwald, 1997) was designed to 
examine the impact of kindergarten restructuring on 16 classes in 16 elementary 
schools on curriculum, professional development and student outcomes. Evaluation 
strategies included focus groups with Bank Street College trainers/facilitators, survey 
responses of program and non-program teachers, observations of training workshops 
at Bank Street College, and ethnographic observations of the kindergarten programs by 
an outside consultant. Quantitative methodologies included the analysis of student 
outcomes on the district kindergarten test and a comparison of student outcomes with a 
control cohort of students in non-New Beginnings classes. Findings based on qualitative 
methods documented the effectiveness for transforming teacher-directed classrooms 
into child-centered ones; for revitalizing teachers through staff development; and 
observed growth in students' social and problem-solving skills. Findings based on 
quantitative methods, and the comparison of achievement outcomes for program and 
control cohorts, showed significant improvement in student outcomes at the end of the 
first year on the district test. 

A second year evaluation prepared by Bank Street College evaluators Fran 
Schwartz and Jonathan Silin focused on the expansion of the program to 20 schools to 
include pre-K to second grade initiatives to strengthen and reinforce developmentally 
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appropriate curriculum before and beyond kindergarten (Schwartz and Silin, 1999) The 
second year qualitative strategies were designed to capture how the expanded 
instructional, staff development, and program focus were changing the contexts of 
instruction, changing teacher thinking, changing classroom life for students, and 
understanding the process of change. These researchers’ observations supported the 
earlier findings showing increases in small group instruction, activity-based learning 
centers, student initiated learning and self reliance. Changes in staff development 
added to first year’s findings by identifying teacher’s needs to learn about the culture of 
the schools and community and experiences of their students. Stages of change were 
identified with shifts from changing language and physical classroom environments , to 
the beginnings of recognition of changes in their own behaviors. 

The third year evaluation also prepared by Bank Street College evaluators 
(Schwartz, Silin and Miserendino, 2000) again used multiple qualitative strategies, 
including in-depth open-ended and structured observation protocols, focus groups with 
early cohort teachers, interviews and surveys that focused on the continuation of 
developments related to staff development, classroom learning, and teacher and 
student interactions as the program moved to include grades pre-K, K, 1 and 2. These 
studies not only provide rich historical perspectives on program implementation and 
impact, but also capture dimensions of change. The findings emphasize the 
solidification of previous gains, noting increased student self reliance and student 
initiated learning, a heightened sense of shared purpose in the work of classroom 
teachers and staff developers, improvement in a shared agenda between New 
Beginnings project and NPS noting the project’s increased responsiveness to the 
District’s new literacy initiatives that are aligned with the New Jersey Core Curriculum 
Content Standards. New Beginnings, in its third year, found a shared agenda in 
collaboration with the district to explore the State Supreme Court whole school reform 
models as they relate to early childhood , and shared work and vision with the district’s 
commitment to improving literacy teaching. Fewer of the earlier concerns, earlier 
tensions and dissonance about serving 2 masters, the district and the New Beginnings 
instructional priorities, appeared to preoccupy teachers. 

Quantitative strategies involving the analysis of student outcome data contributed to 
another component of the third year project evaluation. The Newark Public Schools 
research team conducted a limited follow-up on the progress of students from the first 
year kindergarten cohort using 2 indices to assess student outcomes. These included 
the follow-up and tracking of student outcomes on the district norm referenced test 
given at 2"*^ grade, and on attendance data. The Newark Public Schools research team 
matched the first year kindergarten enrollment data base of 16 classes at 16 schools 
from 1997 with the second grade district test score database in 1999 to compare 
student achievement outcomes on the Stanford 9 norm reference test for both the New 
Beginnings kindergarten cohort and the comparison cohort. The effects of one, two 
and three year supports were examined using analysis of variance and post hoc 
statistics. Attached in Appendix B are tables with findings from these analyses. 

Findings show that students in the first kindergarten cohort, who received all 3 years of 
New Beginnings classroom interventions, attained scores that were significantly higher 
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in reading, math and sub-test skills as compared with students who received less 
intensive and/or interrupted or no program intervention at all. All other comparisons of 
less intensive New Beginnings interventions were not statistically significant. 

Attendance was examined based on comparisons of New Beginnings with control 
group classroom level data. Findings from these analyses were more meaningful to 
examine with the data disaggregated by school and classroom. New Beginnings 
classes at 5 of 12 schools included in the analysis of attendance data showed higher 
attendance rates than control classes using an index of paired comparisons. At 2 
schools, both 1®^ and 2"'^ grade attendance comparisons for the same K cohort students 
were favorable for the New Beginnings classes. These schools were the only K-2 and 
K-4 grade level schools. Additional exploration to interpret this data will be part of the 
continuing work between Bank Street College and the Newark Public Schools. 

Continuing collaborative work will be focussed on understanding the findings 
from multiple perspectives, and will include elaboration of the changes which have 
occurred, the identification of areas of convergence and divergence of evidence, and a 
refocusing of resources to continue progress toward institutionalizing the changes 
necessary for successful Newark Public School reform in early childhood education. 

School Based Health Care Clinics - 

Given the high risk nature of health status of children in the Newark Public 
Schools, the SBHC model currently implemented at 3 elementary schools represents 
innovative partnerships among the Newark Public Schools, the Healthcare Foundation 
of New Jersey, the Saint Barnabas Health Care System, Children’s Hospital of NJ at 
Newark Beth Israel, and the Prudential Foundation. The model is designed to respond 
to every day problems that include early diagnosis, treatment and follow-up for an 
array of healthcare services.. The increasing number of research studies document 
that such nationwide efforts are servicing large numbers of students, are providing 
accessibility and a wide array of services,, and are demonstrating positive outcomes. 
The research literature is clear about the correlation between various social problems 
like poverty, and family problems such as violence and substance abuse, on the ability 
of children to master age appropriate developmental tasks, to achieve academically 
and socially, and to function as productive members of their communities (Morris, 
1991) 

The Newark school based health services model provides a wide array of daily 
physical health services, screenings, education, dental health, and mental health 
services to address a serious gap in services accessible to Newark families. The 
health clinics are staffed by healthcare professionals under the supervision of a 
pediatrician from the Children’s Hospital of New Jersey at the Newark Beth Israel 
Medical Center. In the first year of implementation there were over 2,000 visits with 
an average of 80% parental consent, which is above the national average. The 
collaborative evaluation planning has included the various healthcare institution 
partners, and various departments of the NPS - i.e. Office of Planning, Evaluation and 
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Testing, Office of Community Affairs, Office of Student Information, Office of Student 
Health, and school administrators. Project goals are to ensure that our children are 
able to learn at their maximum potential; that they be healthy learners; and that their 
medical, educational and social needs be met. 

The evaluation design used multiple strategies and data from varied sources 
including quantitative and qualitative clinic utilization data entered on the site-based 
data management system Healthcare on Line at the 3 clinic sites. The design used 
comparison group and trend analysis to compare outcomes for students who do and do 
not receive services, and tracks the changes over time on indicators such as health 
behaviors, attendance and student achievement. Partnership and sharing of resources 
included 1) an ongoing evaluation planning committee composed of partnership 
representatives, 2) a capability to track student level outcomes on health indicators, 3) 
capability to export data with common student identifiers to enable health system data 
to be merged with student outcome data, and 4) cost effective evaluation 
implementation. The first year annual report was a compilation of data reflecting 
program implementation, and survey and focus group responses related to clinic use. 
Compilation and dissemination of a brief first year report was a shared responsibility of 
collaboration partners. Refer to Appendix B for charts that show range of services and 
nature of health problems identified at the three clinics during first start-up year of the 
clinics. Sept. ‘98-May ’99. 

During the first year of the project, the evaluation design developed by the committee 
focussed on the range and quality of services and on implementation of activities 
based largely on the health care management system data. The second year's focus 
will identify changes occurring: 1) at the student level, in school attendance, tardiness, 
selfesteem, attitudes toward school, and in discipline and behavior problems, 2) at the 
school level on indicators such as school climate and parent involvement; an external 
evaluator will provide resources for this evaluation; 3) at the community level, on need 
to address the long range health care issues for children and families, ...i.e., has there 
been a reduction in use of emergency room and use of charity care by families? The 
health care partners have also agreed to fund a survey of families with children 
attending Newark Public schools to identify current access and utilization of school 
based health services in those clinics. Barriers and gaps in community health care 
services for families with children will be identified based on a comparison of survey 
findings from a sample of families using school based services with a control group 
sample. Findings will have broader implications for future policy making about access 
and funding for essential health care services to children and low-income families living 
in Newark. 

21^^ Century Community Learning Centers 

Newark Public Schools were among 99 sites nationally to receive the first wave 
of a federally funded grant to deliver extended-day academic and enrichment programs 
to students at risk of academic failure. The program targets students in grades 6-8 at 9 
district schools to assist them to be healthy, confident, self-directed and responsible. 
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and prepared to successfully negotiate the demands and challenges of high school. The 
NPS goals were adapted from those outlined in the federal legislation to open schools 
beyond school hours through partnerships between school and community based 
organizations, to provide services particularly to middle school youth in a safe, healthy 
and drug-free environment, to improve behavior, school attendance, and academic 
achievement to better prepare them academically and socially to succeed in school. 

The core activities in the Newark grant, now in the second of its three year 
funded cycle are 1) Expanded Learning with an emphasis on New Jersey’s Core 
Curriculum Content Standards and preparation to achieve proficiency of the N.J. State 
Assessment at 8“^ grade; 2) Health Education with emphasis on health, nutrition, conflict 
resolution, self discipline and personal responsibility: and 3) Parental Involvement, with 
emphasis on parent education to their adolescent children in academic, social and 
emotional areas. In addition, each center schedules activities that also reflect 
community needs and interest of parents who attend. 

Each of 9 project sites was paired with a community-based agency or with the 
NPS After-School Youth Development Program. Evaluation Planning included 
collaboration with the Department of Education to define and refine requirements for 
National Annual Reporting which include quantitative and qualitative indicators of 
program and student level outcomes for 99 sites nationally. Partnerships for the 
Newark project included several community based organizations, a project director, 
school and district level administrators and staff, and the the District Office of 
Planning, Evaluation and Testing. District collaboration with one of the community - 
based organizations helped provide technical assistance in developing a site-based 
data management data base for the 9 schools in the project to use for tracking program 
implementation and to assist the sites with recording program evaluation data including 
student ids’, student attendance, activities participated in, and intensity of interventions. 

During the first year of the program, a "case study" was conducted by an external 
consultant and the district research team. The study used ethnographic observations, 
focus group interviews and participant evaluations to profile differently implementing 
models at 5 sites to capture and inform second year program planning, to disseminate 
cross- site information of promising practices as well as to identify problem issues. 

Success stories were shared by administrators as well as by school and CBO staff. 
For example, the impact at one special education school was dramatic. For the first 
time, students with limited learning potential , were able to exercise, to socialize with 
peers, and leave their neighborhoods for recreation. At this site, the principal, project 
staff, and especially parents were enthusiastic, and reported substantial progress in 
students’ confidence, self-direction , and learning of life skills. At this school, students’ 
attendance, motivation, discipline and social skills all showed marked improvement. 
Across sites, and of 295 students surveyed, more than 95% gave the program high 
marks for helping them learn and have fun. 
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The first year case study of 5 schools also provided baseline data on at least one 
measure related to student outcomes. A comparison profile of average monthly 
attendance for 2 years at the 5 schools was constructed. These findings supported 
other qualitative evidence of improved student attendance at participating schools 
during the first year of the program. The table that shows comparison of school 
attendance before and after first year of program implementation is included in 
Appendix B. 

The evaluation design for this program, now in its second year of implementation has 
been complex and multi-faceted. It has included ongoing regular evaluation planning 
meetings with the district research team, with the project director/manager, with each 
site's community based organization coordinator and each of the 9 sites’ school liaisons. 
Work sessions were scheduled to 1 ) prioritize and streamline the data collection 
process to enable accurate and reasonable tracking of program implementation; 2) to 
align data collection with national annual reporting requirements: 3) provide various 
forms of technical assistance to help facilitate data collection for the program staff; 4) 
troubleshoot the practical obstacles which impede the evaluation process; 5) dialogue 
and communicate (at various levels including with funders) about the kinds of data and 
outcome measures on which the program would be evaluated; and 6) clarify timelines, 
reporting protocols, and the varied responsibilities of partners in the evaluation process. 



In summary, the following issues highlight the contributions and benefits which 
collaborative partnerships and the mixed method paradigms can offer to the systemic 
reform process: 

Collaboration promotes shared understanding of different concerns and 
expectations of program objectives, and helps to identify unmet needs, 
which are important to various stakeholders. 

Principals’ Leadership Institute (PLl) survey findings and collaboration 
identified and emphasized areas of congruity and incongruity between the 
various professional development models on the one hand, and the principals' 
expectations and perception of needs based on their daily experience of school 
management, on the other. The data and analysis also allowed for the 
elaboration of the efficacy of models grounded in very different theoretical 
premises. The intensive qualitative data gathered from the interviews with the 
universities provided important qualitative contextual information, which helped to 
clarify the concerns raised by the principals. 

New Beginnings Kindergarten observation and focus group strategies 
conducted by Bank Street College evaluators elaborated upon instructional 
objectives and behavioral change. The district research team analysis of 
student achievement and attendance data facilitated cohort tracking and 
elaborated upon student and class outcomes. The collaborative design focused 
exploration and understanding from different perspectives and on different 
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outcome objectives. One finding identified by the qualitative researchers in third 
year study was a reduction in tension experienced by teachers as “serving 2 
masters", both Bank Street and the Newark Public Schools ”, seen earlier on as 
exemplifying competing objectives. Findings from the second and third year 
studies elaborate upon the stages of change and shifts over time toward a) 
increasing congruity for teachers; b) increasing responsiveness to the District’s 
new literacy initiatives; and c) improving collaboration on early childhood 
methods and pedagogy as they relate to the new district reform process. 

SBHC collaborative planning included healthcare stakeholders, and 
several central office departments including Community Affairs, Health Services, 
Student Information, Planning, Evaluation and Testing and the principals of 3 
participating schools. Different partners provided varied sources and resources 
for data collection and data management, and helped participate in choice of 
research methods and design of study. Healthcare stakeholders provided 
additional resources for conducting larger scale needs assessment of family 
health care practices to identify gaps in the community 's resources to provide 
needed services and to inform future planning. 

Collaboration helps clarify division of labor, contributes to the 
development of trust, and fosters the evolution of models that are 
practically feasible as well as methodologically sound. 

The collaborative process serves to legitimize voices and overall 
commitment to the evaluation process. On the 21st Century Community 
Learning Centers program, collaboration with DOE on the annual report of data 
helped problem solve the complexity of tracking program implementation data, 
student outcomes, and issues of uniformity of reporting outcomes across sites 
within the District, and for the DOE, across sites nationally. 

Project GRAD is an example of positive collaboration between external 
evaluators, program managers, and the District Testing Office, which assisted 
and facilitated the evolution of multi-cohort evaluation paradigms. The choice of 
cohort and trend analysis as options to longitudinal tracking of individual student 
outcomes provided interim solutions when limitations in student information 
system capability existed. As exemplified by a number of the evaluation models, 
including Project GRAD, SBHC, and New Beginnings, collaborative partnerships 
facilitated the division of labor and resources necessary to conduct labor 
intensive qualitative research as part of more comprehensive evaluation designs. 

Collaborations strengthen the resources and infrastructures to conduct 
multi-method evaluation paradigms and to meet multiple reporting 
requirements of program outcomes for varied audiences. 

On all projects illustrated here, partners collaborated to identify resources 
needed to track program implementation, and to collect and compile varied data 
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for summative assessment. Collaboration also enabled everyone to assess 
availability of partnership resources to realistically align evaluation design with 
the priorities of program objectives and funding requirements. 

Multiple data management systems that become a part of the joint 
ventures are examples of the way programs can better manage and accomplish 
complex reporting and evaluation requirements. For example, the school-based 
clinics dedicated health information management system, which records student 
health and clinic usage information, also provided "data fields" for linkages with 
district student and school data. This enabled us to combine program usage with 
student outcome information. 

Collaboration promotes the reasonable planning of project timelines 

Varied expectations, which surround the requirements of measuring and 
reporting program outcomes are aired and discussed among the various 
stakeholders. Critical evaluation questions to address include - What is 
reasonable to expect in the short run? What is to be expected in the long run? 
what do various stakeholders want to know? What resources can various 
stakeholders provide to the evaluation effort? 

Staging the evaluation is important to everyone. Stage 1 includes tracking 
the implementation of services; stage 2 includes analysis of data from varied 
sources related to outcomes expected in the short term ; stage 3 includes 
combining long range findings from various perspectives that address changes 
in attitudes and behavior over time. The shared responsibility of collaborative 
planning fosters buy-in and commitment based on apportioned responsibility and 
reasonable timelines. The collection and sharing of information based on project 
timelines facilitates its use for various purposes and audiences, provides interim 
feedback for project management purposes, and more comprehensively informs 
the interpretation of findings. 

Collaboration fosters dissemination of study findings to broader 
constituencies. 

Reporting supports sustainability of promising and "best practice models" 
of grant funded initiatives, provides access to additional funding sources, 
enables feedback and decision making on incorporation of models into District 
reform planning. For example, the first year annual report of the School Based 
Health Clinic Project included data to document the progress and beginning 
successes of the program partnership, noting the frequency and nature of clinic 
usage, services provided and student needs which were met. Compilation and 
dissemination of a brief first year report was a shared responsibility of 
collaboration partners,and provided important feedback to a variety of 
stakeholders with serious interest in seeing the model replicated at additional 
schools. 
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Multiple methods provide different lenses from varied sources that 
contribute to meaningful and informed outcomes. Multiple methods serve 
to better inform the contexts within which educational change occurs. 

The qualitative and quantitative approaches in the PLI study served to 
better inform the complexities that exist between the practical realities and 
needs of practicing administrators as discussed earlier, and the theoretical 
assumptions which undergirded the universities' leadership development models. 

New Beginnings findings from second and third year studies using a 
variety of qualitative methods helped to illustrate not only changes in practices 
but stages of change in teaching roles and instructional practices. Other 
findings identified systemic changes such as shifts to a shared agenda between 
New Beginnings project and the NPS new early childhood literacy initiatives. 
Quantitative strategies provided evidence to support pedagogical beliefs of the 
New Beginnings model, and of practice based research, that 3 years of sustained 
program interventions resulted in significantly higher student outcomes when 
compared with outcomes of students who received less intensive, interrupted, or 
no New Beginnings program interventions at all. 

Multiple methods overall, help to strengthen evaluation design. 

Varied sources and methods in all the studies discussed served to 
strengthen overall design and reliability of information and findings. In addition, 
the use of cohort and trend analysis provided alternative quantitative strategies 
that helped to resolve current limitations in capabilities of student information 
management systems to longitudinally track student outcomes on multiple 
indicators. 

Results and Conclusions 

The evaluations discussed in this paper illustrate how exploratory and confirmatory 
investigations using multiple techniques can strengthen design, analysis and inference, 
improve conceptual clarity, flexibility, comprehensiveness, scientific rigor and efficiency. 
They reflect the complexity of today's comprehensive school reform goals. As mixed 
methods help to provide varied perspectives, strengthen evaluation design, and 
reconcile methodological concerns about scientific rigor, multiple voices and 
collaborative partnerships help to disseminate findings that inform varied audiences, 
document different level program outcomes, make recommendations and identify needs 
to refocus strategic planning for the future. 

In the past, evaluation models for education programs tended to be narrow, and 
to mirror schools as closed communities. Evaluation paradigms are changing and are 
looking very different. As this paper suggests, they reflect the changing nature of 
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educational reform, which are more comprehensive in scope and involve broader-based 
participation. The comprehensive nature of collaborations foster more inclusive buy-ins 
for ongoing strategic planning, improve access to multiple data sources that inform 
program goals, school and district planning, as well as student outcomes, and are 
best served by multi-method evaluation strategies which are more cost effective. 
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Resource List of Program Contacts 



21®* Century Community Learning Centers 
Elnardo Webster, Project Manager 
21®* Century Community Learning Centers 
Newark Public Schools 
2 Cedar Street 
Newark. N.J. 07102 
ewebster@nps.k1 2.nj.us 

New Beginnings - Newark 
Carol Lippman 
Director, New Beginnings 
Bank Street College of Education 
610 West 112*'’ St 
New York. N.Y. 10025-1120 
(212) 875-4504 

Principals' Leadership Institute 
Wayne Turner, Director 
Staff Development/lnstructional 
Newark Public Schools 
2 Cedar Street 
Newark, N. J. 07102 
wturner@NPS.k1 2.nj.us 



PROJECT GRAD NEWARK 

Tycene Hicks-Edd, Director 
Two Gateway Center, 15**’ fir 
Newark. N.J. 07102 
(973) 645-5225 
thicks@lucent.com 



School Based Health Clinics - Newark 
Armando Medina, Special Assistant, 
Office of Community Development 
Newark Public Schools 
2 Cedar Street 
Newark. N.J. 07102 
amedina@nps.k12.nj.us 
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NEWARK COLLABORATION MODEL 
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MULTIPLE METHOD PARADIGM 




Other School Level Outcomes 
Student Achievement 
Attitudes 
Behaviors 
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APPENDIX B 



PRINCIPALS’ LEADERSHIP INSTITUTE EVALUATION 

TABLES 

RIDIT VALUES FOR MID-YEAR AND END OF SECOND YEAR SURVEYS * 



AREA 


MID-YEAR 


END OF YEAR 


Need for greater relevance 


.50 


.48 


Knowledge acquired is practical and 
useful 


.58* 


.68* 


Knowledge acquired is easy to 
implement 


.56 


.39* 


■ — 

Collegiality exists between college 
members 


.48 


.55* 


Universities have much to offer 


.67* 


.61* 


Coordination needed 


.35* 


.34* 


Follow-up support is adequate 


.71* 


.59* 


Mentoring support is valuable 


.69* 


.58* 


Effectiveness as a leader has improved 


.64* 


.55 



Note: N for end of first year = 80, mid-year surveys = 1 08, end of second year survey = 72. * Z values 
statistically significant based on the formula (r - .50/se r). 

Ridit analysis was used to compare the data obtained from the three sets of surveys administered 
to principals and vice-principals (Fleiss, 1981). Because the samples were not truly matched (due 
to attrition and replacement), an analysis which allowed us to use the first set of data as a 
r 0 f 0 i- 0 nce point for understanding changes in the subsequent sets was considered to be the most 
appropriate. From the first set of data we estimated for each scaled question, the proportion of all 
individuals with an attitude score falling at or below the midpoint of each scale value (scale values 
W 0 r 0 1-5 with 1 representing strongly disagree, and 5 strongly agree). Ridit values were then 
calculated for each scale value for a given question. Based on the distribution of the responses 
from the subsequent interviews, a mean ridit value was calculated for each question. If the mean 
ridit for a question on the subsequent interviews was greater than .50, we inferred that a randomly 
selected administrator from the subsequent surveys held a more unfavorable attitude, and 
conversely. This analysis also allowed us to chart the consistency in attitudes and feelings over 
the two academic years. This was accomplished by comparing the ridit values from the second 
and third surveys to the reference point. Z values based on the standard error of the ridits were 
calculated and tested for statistical significance. 



’ Reprinted from (Walker, Mitchel and Turner, 1999) 
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New Beginnings Kindergarten Evaluation: T Test and Post Hoc Analyses 



Table 2 

Kindergarten Cohort: Differences in Total Reading and Total Math Scores 





TOTAL READING 


TOTAL MATH 




Mean NCE 


Mean 

Difference 


Standard 

Deviation 


'Mean NCE 


Mean 

Difference 


Standard 

Deviation 
















New Beginnings 
Kindergarten Cohort 


47.5 

(N=I74) 


3,5* 


18,8 


51.0 

(N=17I) 


4,2* 


22,9 


Control** 


44.0 

(N=460) 


21,7 


46.8 

(N^444) 


24.2 1 



* The mean differences are significant at .05 level. 

Levene’s Test of variances was conducted to control for the sample size differences 



Table 3 

Kindergarten Cohort: Differences in Total Reading and Total Math Scores Disaggregated by 

Grade Levels of Support 



* : py :, ? - -k, - . 


READLNG 


MATHEMATICS 


Mean NCE i Mean Difference \ 


MeanNCE: 


Mean Difference 












K, Gl & G2 
Vs 

K&G2 


55.8 

SD=17.8 

(N=73) 


12.6* 


59.2 

• SD=22,8 

(N-7J) 


11.3* 


43.2 

SD=12.2 
1 (1^=26) 




47.9 

SD=15.6 

(N=26) 


K, Gl & G2 
Vs 

K&Gl 


iiii 

SD=17.8 

(N=73) 


18.8* 


59^2 

SD=22.8 




16.4* 


37.0 

SD=17,3 

(N=40) 


42.8 

SD=26.5 

fN=38) 


K, Gl & G2 
Vs 

Konly 


55.8 

SD=17.8 
1 fN=73) 


10.3* 


59.2 

SD=22.8 

(N=71) 


13.5* 


45.5 

SD=19.5 

(N=35) 


45.7 

SD=18.3 

fN=36) 


K, Gl & G2 
Vs 

1 Control 


55.8 

SD=17.8 

(N=73) 

1 

SD=21.8 
1 (N--460) 


11.8* 


59.2 

SD=22.8 

(N=7I) 

46.8 

SD=24.2 

(N^444) 


12.4* 



ANOVA with Post Hoc (Fischer’s LSD) 

* Significant at ,05 level in reading and math. 
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New Beginnings Kindergarten Evaluation 
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COMPREHENSIVE SCHOOL-BASED HEALTH CLINICS: THE NEWARK MODEL^ 



Given the health status of the children in the Newark public schools, a school-based health 
clinic model was designed to specifically respond to their needs. In January 1998, the first full- 
service school-based health clinic was opened at the George Washington Carver/Bruce Street 
Elementary School in Newark, New Jersey. The Carver Clinic represents an innovative and 
effective three-way partnership among the Newark Public Schools, the Saint Barnabas Health 
Care System and The Healthcare Foundation of New Jersey. A multi-disciplinary team of 
healthcare providers works cooperatively with the school nurses to enhance existing school 
health services. The school nurse serves as the gatekeeper, referring students to the 
appropriate caregiver. Comprised of 1,000 square feet, the clinic was designed to be 
accessible and very “child friendly” with lots of children’s art covering the walls. 



The health clinic is staffed by healthcare professionals from The Children’s Hospital of New 
Jersey at the Newark Beth Israel Medical Center, under the supervision of a pediatrician. A 
full-time pediatric advanced nurse practitioner providers primary care including diagnoses, 
screenings, treatment and health education. A full-time Master’s level social worker conducts 
one-on-one and group counseling services. He/she works closely with the school’s child study 
team and pupil assistance committee. A dentist provides students with annual dental exams, x- 
rays, topical fluoride treatments and sealants (as necessary). While not the first school-based 
clinic system in America, the Newark Model is unique in that medical, dental and emotional care 
are combined in one school location. The success of the Carver Clinic over the past 18 months 
has led to a replication of this model in two additional Newark public schools: the Quitman 
Street Elementary School and the Dayton Street Elementary School. The Prudential 
Foundation, which has supported the community school initiative at the Quitman Street 
Elementary School, has joined as a partner with The Healthcare Foundation of New Jersey to 
fund the Quitman Street school-based health clinic. 

ACCOMPLISHMENTS TO DATE 

From September 1998 to May 1999, there were 2,740 visits at the three school-based clinics. 
The G.W. Carver Clinic is fully operational, providing primary health, dental care and mental 
health services. The Dayton Street and Quitman Street Clinics are currently providing mental 
health and dental services and will be fully operational in June 1999. Health clinic services are 
available to all children whose parents/guardians have signed consent forms. The national 
average for parent consent permitting use of school-based health care facilities is 67%. 
Newark’s high average of 80% speaks to the confidence that parents have with regard to these 
clinics. 



1 Abstracted from Report on Newark’s School-Based Health Clinics June 10, 1999. A collaboration of 
The Healthcare Foundation of New Jersey, Children’s Hospital of N.J., Newark Beth Israel Medical 
Center, Saint Barnabas, Healthcare System, Newark Public Schools, The Prudential Foundation. 
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CLINIC VISITS: 



The following table shows the number of parent consents at each school and the number of visits 
for available health care services: 









CLINIC VISITS 








SCHOOLS 


# CHILDREN 


% CONSENTS 


# DENTAL 


# SOCIAL 


# MEDICAL 


TOTAL # OF 
VISITS 


G.W. Carver 


1200 


72% - 864 


641 


568 


488 


1697 


Dayton St. 


411 


97% -399 


324 


514 


- 


838 


Quitman St. 


643 


81% -520 


- 


205 


- 


205 


TOTALS 


2254 


80% - 1783 


965 


1287 


488 


2740 



REFERRALS FOR ADDITIONAL HEALTH SERVICES: 



If the healthcare provider identifies the need for more advanced therapeutic services, the children 
are given referrals to off-site medical care. Clinic staff provides follow-up with students’ parent 
and physicians to ensure that needed medical services were rendered. As shown below, significant 
referrals were given to children in all service categories. 





VISITS AND REFERRALS 






# 


# 


# 


# 




# 


TOTAL # OF 


SCHOOLS 


DENTAL 


REFERRAL 


SOCIAL 


REFERRAL 


MEDICAL 


REFERRAL 


REFERRALS 


G.W. Carver 


641 


40 


568 


88 


488 


81 


209 


Dayton St. 


324 


84 


514 


19 


- 


- 


103 


Quitman St. 


- 


- 


205 


38 


- 


- 


38 


TOTALS 


965 


124 


1287 


145 


488 


81 


350 



EDUCATIONAL PRESENTATIONS: 



All children in the school receive classroom-based dental education, disease prevention 
instruction and health kits. The following table details the number of classroom-based 
and community educational presentations made by clinic staff from the three clinics. 





EDUCATIONAL PRESENTATIONS 






# 


# 




# 




it 


SCHOOLS 


DENTAL 


PARTICIPANTS 


MEDICAL 


PARTICIPANTS 


SOCIAL 


PARTICIPANTS 


G.W. Carver 


55 


1155 


13 


514 


6 


122 


Dayton St. 


20 


420 


0 


0 


19 


603 


Quitman St. 


9 


397 


1 


30 


8 


146 


TOTALS 


84 


1972 


14 


544 


33 


871 
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FEEDBACK FROM STUDENTS, TEACHERS AND PRINCIPALS 



A kindergarten teacher at G.W. Carver Elementary School, wrote to the Clinic Coordinator, “My 
kindergarten class and I have benefited so much from the health clinic. The children have had 
excellent instruction in personal hygiene. The social worker is very skilled and she has helped 
my children tremendously. Thank you for the innovative ways that you have designed this 
program. It is really terrific to see that children can be helped by caring people who assist with 
their needs so quickly and so caringly". 

Studies of urban school-based health clinics published in peer reviewed journals indicate that 
students enrolled at schools with clinics were more likely to visit any healthcare provider 
compared to adolescents in the general population.^ In another study, students who accessed 
clinical social workers at school-based clinics reported significant declines in depression and 
improvements in self-concept from pre-and post-intervention. Fewer students attending school- 
based health clinics reported considering suicide compared to what would haVe been expected 
if they followed the trend of urban youths nationally. ^ 

Though it’s too early to assess the long-term benefits of these new health facilities, a series of 
focus groups were held with students and surveys administered to teachers to understand the 
perceptions connected with the use and validity of their respective school-based health clinic. 

Focus groups were performed with 75 students (grades 3-8) from the three Newark schools to 
learn how the services were viewed. Over 80% of the students who participated in these focus 
groups reported that they like using clinic services at their schools. All of the students were 
familiar with the services and were comfortable with the staff and with the facility. Students 
indicated that they experienced the following: 



♦ 21% reported having asthma 

♦ 80% reported having headaches 

♦ 85% reported having stomachaches 

♦ 27% reported feeling tired 

♦ 79% reported feeling worried 

♦ 53% reported feeling sad 



32% reported low self-esteem 

59% reported failing classes 

45% reported missing a lotof days from school 

43% reported not feeling safe at school 

72% reported feeling angry 

63% reported feeling frightened 



' Anglin, TM; Naylor, KE; and Kaplan, DW. Compared utilization of medical, mental health and substance 
abuse services among students who use school-based health centers with students who used traditional 
sources of care. The researchers analyzed diagnoses from 27,886 visits to three school-based health 
centers; 1996. 

^ Kisker, EE and Brown, RS. Compared students at 19 Robert Wood Johnston Foundation-sponsored 
school-based health centers with a national representative sample of urban youth; 1996. 
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TEACHERS FEEDBACK 



The following chart highlights the surveys completed by a sampling of 59 teachers form the 
three Newark clinics: 



SURVEY 

QUESTION 


G.W. CARVER 
(N-14) 


DAYTON STREET 

(Nz22) 


QUITMAN STREET 
(N=23) 


How has the school-based health 
clinic impacted students this 
year? 


♦ education and care of the 
whole child - physical and 
emotional needs 

♦ social worker participated on 
parent resource committee 

♦ reduced absences 

♦ dental services, eye exams 

♦ great to have services our 
students need right here at school 

♦ monitor health problems 


♦ dental checkup have been 
very helpful; children had 
opportunity to experience 
having their teeth cleaned 

♦ counseling services for 
students with emotional and 
behavioral problems 

♦ eye and ear exams 


♦ counseling services; 
intervention during angry 
disputes 

♦ social worker 

represents positive role model for 
my students 

♦ immunization checks 


What services have been most 
helpful? 


♦ diagnosis of specific illnesses 

♦ the social worker has been 
most helpful and always follows 
through with all problem brought 
to her attention 

♦ dental health services 

♦ eye exams and hearing 
exams that students othenA^ise 
don’t receive 

♦ checking ears with 
typanogram; removal of ear wax 


4 social worker’s individual and 
group work has been very helpful 
♦ conflict resolution/anger 
management; peer groups 


♦ social worker and nurse have 
provided professional support for 
students in emotional crisis 

♦ the nurse’s educational 
program on tuberculosis 



The teachers are familiar with and strongly support clinic services based upon their answers to a number 
of open-ended questions about the services provided. Some comments include: 

“Students now have a place to get needed care and still remain in school”. 

“The school-based health clinic has positively affected the education and care of the whole child, both the 
physical and emotional needs. We are seeing fewer absences. The dental and eye exams have shown 
positive results”. 

“The social worker has been quite helpful with many of my students who are having emotional and 
behavioral problems. She always follows through in a timely and thorough manner with all situations 
brought to her attention”. 

“It is great to have the services our students need right here at school. This way we can be sure they are 
properly taken care of’. 

“Please continue with the health clinic. Its services are invaluable to the total well being of our 
children”. 

“Clinic staff represent positive role models for my students”. 
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PRINCIPAL FEEDBACK 



All three principals in the schools with health clinics have noted significant benefits to their 
students, school and community. 

The principal at Quitman Street said, “While we are not yet a full service health clinic, during this past 
school year we were pleased to phase in some components such as child/ family counseling and a yearly 
scheduled vision and dental screening. These services have encouraged us to work side by side with the 
parents, teachers and community to ensure that children are given every chance to succeed. By locating 
the clinic in our school, health problems can be caught and addressed before they become more serious . 

The principal at Dayton Street School expressed that the clinic is a valuable part of her school. The 
creation f the health clinic at Dayton now provides our children with a consistent and thorough health care 
program that was not available to the students in the past due to limited financial and medical/dental 
resources. The establishment of our strong and meaningful partnership will impact positively on our 
children. For the children of this school, having healthy body, will contribute to having a healthy 
educated mind”. 

The principal of the G.W. Carver School stressed that the whole child gains when a school and health 
clinic work together and her hope that this concept/model is practiced in as many schools as possible. 
Furthermore she noted that having a clinic at the school has enabled “health needs to be responded to in a 
timely manner, decreasing the time students may have to miss school because of health concerns”. She 
goes on to say that with the availability of medical and mental health services early in the morning and 
after-school, “the hours of operation demonstrate the belief that the child’s needs come first. The 
program responds to the whole child thus ensuring greater academic success for our children. 

CONCLUSION 

The three existing school-based health clinics provide high quality, immediate primary and 
preventive health services to every child with parental consent regardless of his/her insurance 
status or ability to pay. On-site school-based health clinics are working in Newark. A 
population of children beset with enormous environmental and health disadvantages are now 
having their medical, dental and emotional problems identified and treated earlier. Children are 
smiling proudly, showing off their recently cleaned and examined teeth. Youngsters with 
emotion needs are talking to sensitive caregivers about their difficulties. Over 2,000 children 
have visited the three clinics for a wide range of health services. Students, parents, teachers 
and administrators concur - these clinics are essential for the positive growth and development 
of Newark’s children. 

“Foundation, healthcare and public school resources are now allied to help 
these young people become healthy learners. The Healthcare Foundation of 
New Jersey is thrilled to be working with all of these organizations to provide a 
healthier Newark now and into the millennium". 
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Mark S. Hochberg, M.D. President 
The Healthcare Foundation of New Jersey 
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